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ABS'^RAC^' 

This reoort describes a 1— week institute designeu to 
uparade research and evaluation skills of persons associated with PL 
39 l'IQ Title I and *^itle III projects. The institute’s objectives were 
to illustrate and provide experience in: (1) operationalizing 

obi 0 ctives of instructional programs; (2) designing projects with^^ 
appropriate controls to allow valid observations of change; and (o) 
constructing and applying observational tools, besides> standa^-dize 
tests, appropriate -^or assessing outcomes of instructional programs. 
Of^the 23 participants, 70^^ evaluated their change in ability by 
respondina' to a postsession Questionnaire. In general, respondents 
indicated "that they had either "improved somewhat" or "improved 
greatly, " their ability to deal with the topics of the institute. The 
appendix includes copies of the institute program, a list of 
institute participants, and pre” and postsession questionnaires 
administered to the participants. (JH) 
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SUMMARY 



This report describes an institute designed to upgrade research and 
evaluation skills of PL 89-10 Title I and Title III personnel. The institute’s 
objectives were to illustrate and provide experience in: a) operationalizing 

objectives of instructional programs, b) designing projects so as to provide 
controls and avoiding sources of invalidity so that meaningful observations 
of change can be made, and c) constructing and applying observational tools 
beside standardized tests, appropriate for assessing outcomes of instructional 

programs . 

The institute convened on November 20, 1968 with two-and-a-balf hour 
meetings in the morning, two-and-a-half hours in the afternoon and an hour- 
and-a-half of tutoring in the evening. The meetings continued, with the 
exception of Saturday afternoon and Sunday, through November 26, 1968, 

Topics dealt with were; basic, nonexperimental and quasi-experimental designs 
for research— 'their limitations and sources of invalidity; operationalizing 
outcomes of research problems; developing ani applying measuring tools for 
specific project outcomes; applying unobtrusive measures and nonreactive var- 
iables in analysing project outcomes; and item sampling techniques appropriate 
to assessing teaching outcomes while economizing on testing time. 

The program was evaluated by mailing a self report questionnaire to each 
participant after the institute was completed. Responses to the questionnaire 
indicated a general belief that participants had clearly upgraded their skills 
in research design and in the evaluation of the impact of instructional programs. 



FINAL REPC51T 

INSTITUTE IN RESEARC2I DESIGN AND EVALUATION 

Introduction 

The value of Title I and Title III projects under PL 89-10 depends 
upon the adequacy of their design and the effectiveness of their evaluation. ' 
However, Guba (1967) after surveying for the U.S, Office of Education the 
evaluation schemes involved in Title III proposals concluded that most 
proposal writers are still uninformed about emergent meanings and uses of 
evaluation, and seem unaware of the criteria that might be applied to an 
evaluation design to determine whether it is appropriate or inappropriate. 

As a result, Guba believed, the typical design is inferior and likely to 

lead to Invalid or unreliable conclusions. 

Therefore, it was the purpose of this institute to provide experiences 
which would upgrade the research and evaluation skills of Title I and Title 
III personnel. Specifically, the institute proposed to illustrate and provide 
experience in a) operationalizing objectives of instructional programs, b) 
designing projects so as to provide controls and avoid sources of invalidity 
so that meaningful observations of change can be made, and c) constructing 
and applying observational tools, beside standardized tests, appropriate 
for evaluating outcomes specific to given instructional programs. 

Methods 

Announcements of the institute were sent to all school districts in 
Indiana and to all federal project supervisers in the states adjacent to 
Indiana. From this list of applicants responding to this announcement, 
twenty-five persons were selected. Criteria were: 

1. The applicant’s supervisor certified that the applicant was 
contractually committed to carrying out a phase of a Title I 

or Title III project. 
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The applicant had at least two years of school experience, 

3, Preference was given to persons less than 50 years of age, 

4, On the basis of reported training in research and evaluation 
the applicant could prpfit considerably from an institute 
covering the topics proposed for instruction. (Appendix A) 

Twenty-three of the twenty-five applicants selected actually appeared 
for the institute. 

Instruction followed the typical pattern of two-a-a-half-hour sessions 
each morning and again each afternoon. Problem assignments were given each 
day and an hour-and-a-half tutoring session was provided each evening to 
assist with these assignments. The outline of the sessions is given in 
Appendix B, Procedures were developed at the ’’rule of thumb” level due 
to the background of the participants and the compressed nature of instruction. 

When participants had returned to their homes a questionnaire was sent 
to them so that they might evaluate their increase in proficiency in topics 
dealt with in the institute. 

Results 

A summary of the results, item by item, of the questionnaire is given 
in Appendix C, These data are based on a 70 per cent reply to the document. 

In general, participants preferred the categories ^improved somewhat” and 
’’improved greatly” to describe the change in their ability to deal with the 
topics of the institute, 

A number of letters have been received from participants expressing 
gratitude for the opportunity to participate in the institute. They have 
also suggested that institutes be proposed in sequence so that successively 
complex topics could be dealt with, with intervening experience on the Job, 
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Some personal observations of the instructional staff are also relevant 
here. The consensus of staff opinion was that the group was enthusiastic 
about the instruction. The requests for faculty time outside scheduled 
sessions indicated that the participants were considerably interested in 
the ideas of the institute and wanted to explore them further. Continuing 
correspondence with participants again reveals this to be true. 

The homogeneity of background, due to the selection process, contributed 
materially to the progress of the group. This condition reduced the need 
for individual instruction and increased the impact of group sessions. 

The bringing together of school research personnel in itself was a 
definite advantage to them. In the local setting many districts do not have 
enough persons in research for personnel to conduct a penetrating dialogue. 

At the institute common problems were explored both within sessions and outside 
of scheduled sessions. This alone was a major benefit of the institute. 

Conclusions 

A short (one week) institute providing Intensive instruction in research 
design and evaluation techniques can be effective in upgrading the research 
skills of Title I and Title III personnel. If the skills of these people 
can be increased in this short a period, great promise appears in repeat 
of several short term programs, each dealing with a successively complex 
topic. 

It is proposed that short institutes such as represented here be developed 
in sequence with planned on the job experiences intervening between Institutes. 
During the course of a year this procedure would also accommodate persons with 
different levels of skill in research design and evaluation, and would bring 
the base level of ability up across many federal projects. 
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DIIUSCTIONS: Some of the terms ir the following list may be familiar to 

you“~others may not* No one is expected to be familiar with all the 
terms* For planning purposes in the Institute, it would be helpful if 
you would react to each term by using the following scheme: 

1* I could give a brief extemporaneous lecture on the concept 
involved, 

2, I could discuss the concept after a little "brush-up « 

3, Sounds familiar, maybe I could follow lectures which provided 
a rapid i«view, 

4, Sorry, I’ve heard of it but it’s pretty foggy to me. 

5, I never heard of it. 



normal curve 

criterion— related validity 
construct validity 
split-half reliability 
stability 

equal-appearing interval 

Lihert scale 

arithmetic mean 

standard error of the mean 

standard deviation 

confidence interval 

variance 

chi square 

z-score 

two-tailed test 
t-test 

level of significance 
predictor variable 
parallel products 
principle of most squares 



covariance 

matched pairs 

product-moment 

regression coefficient 

correlation coefficient 

cross products 

standard error of estimate 

intercept 

slope 

principle of least squares 

homoscedasticity 

linearity 

predicted score 

criterion variable 

variance accounted for 

partial cor ion coefficient 

regression equation 

expectancy table 

degression coefficient 



I especially need information on , 

PLEASE RETURN THIS COIvIPLETED FORf.l BY RETURN I.IAIL IN THE ENVELOPE PROVIDED 

Return of this questionnaire is an expression of my acceptance 
of an appointment in the Institute, Must by postmarked before 
November 11, 



Signature 
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Program 





Institute on ^esearcn Design and Evaluation 


November 20 

9:30 a*m. 


Registration 


10:00 a*ra. 


Review of descriptive statistics through correlation 1.2, 1.3 

4.1, 4.2 


1:00 p.m. 


Probability, confidence limits, simple analysis of 2.1, 3.3 

3 5 

variance * 


7:00 p.m. 


Tutor Session (in dormitory) 


November 21 

9:00 a«m. 


Developing measuring devices for assessing unique 
outcomes of projects; The Thurstone Scale 
(read 4.11 before class) 


1:30 p.m. 


Outline of research designs appropriate for applied 
research, . sources of invalidity, possible conclusions 


7:00 p.m. 


Tutor Session 


November 22 

9:00 a.m. 


Problem identification, definition, assessment 


1:30 p.m. 


Developing measuring devices for assessing unique out- 
comes of projects: The Likert Scale, Q—sort 


7:00 p.m. 


Tutor Session 


November 23 

9:00 a.m. 


Complex experimental design and analysis I 2.2, 2.4 


November 24 


HO SESSIONS 


November 25 

9:00 a.m. 


Complex experimental designs and analysis II 2.4, 4.9 


1:30 p.m. 


Likert Scale and Q-sort (cont.) 
Unobtrusive measures 
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November 26 



9:30 a.m, Heasuring Change 

1:00 p.m. Special Topics 
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INSTITUTE ON RESEARCH AND EVALUATION 
November 20-26, 196S 



DIRECTIONS ; A list of tasks a researcher iza.y be called upon to do is 
provided below. You are not here asked to do any of the problems pre- 
sented, but rather to rate each task on a five point scale as follows, 
”As a result of the recent Institute on Research Design and Evaluation 
ray confidence in ray ability to perform the following tasks has;” 

1, improved fantastically 

2, improved greatly 

3, improved somewhat 

4, improved only slightly 

5, not improved at all 



2,50 * 1, Organize the necessary procedures for obtaining test results 
following the item-sampling technique, 

^>37 2, Compute estimates of a school district mean and standard devia- 

' tion from item sampling data, 

2,67 3^ State why some possible sources of invalidity are not charac- 

teristics of the institutional cycle research design (e,g, , 
history, mortality), 

2,75 4, State why maturation is a possible source of invalidity in 

the institutional cycle research design. 



Item ^ refers to the following problem . 

Assume that the data in the table below represents learning scores of 
36 randomly chosen twelve-year-old boys with IQ*s between 95 and 105, 
The learning was conducted under three methods of instruction (lecture, 
discussion, and demonstration) and by three randomly selected teachers. 
Each teacher used each method on a different group of four boys. 





Teaching Method 


Teacher 


I 


II 


III 




6 


11 


8 


A 


8 


10 


10 


/I 


9 


13 


11 




11 


14 


12 




9 


14 


7 


6 


12 


14 


7 


12 


15 


8 




13 


15 


12 




4 


9 


11 




4 


8 


13 


c 


5 


11 


13 




9 


12 


14 



^>75 5a, Deteimiine the sources of variation, 

2,67 5b, Test the implied hypotheses using appropriate F-ratios, 
2,50 5c, Compute the total sum of squares. 



*Median rating by participants. 
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Item G refers to the following situation . 

The following experiment involved eight (S) classrooms. Four of the eight 
classrooms involved in the experiment are from your own school district 
(Sj) and four from an adjacent school district (S2). Fach classroom was 
administered three types of feedl^ck on an Fnglish composition — positive, 
negative, none. Suppose that each class was split into 9 subgroups at 
random and that three subgroups were given feedback independently under 
each of the three types. Thus, three observations, Xi, X2, and X3, on 
the single dependent variable of subsequent test performance exist for 
each classroom. This is illustrated below: 



School 

District 


Classroom 


r 

Type of Feedbacl 




Positive 


Negative 


None 




1 


Xl, X2, X3 






Your 


2 








Own 


3 










4 










5 








An 


. 6 








Adjacent 


7 










8 









2,35 6a, Determine the degrees of freedom for each source of variation, 
2.75 6b. Test the implied hypotheses using appropriate F- ratios. 

2,67 6c. Write conclusions based upon the F-ratios. 



3,50 7 . Interpret the findings from a single classification, analysis 

of covariance technique, 

3,10 8, State a good reason for using a covariate in an analysis (i.e,, 

why use analysis of covariance). 



Item 9 refers to the follov/ing situation . 

A project proposes to pretest a group of children in reading and then 
provide a specialized training program. At the end of the program a 
posttest will be given. Gain scores (posttest-pretest) will be used to 
assess the impact of the program. 



2,41 9a, Tell why raw gain scores are often misleading. 
2.35 9b, Propose a method of correcting rav? gains. 
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2,67 10, Describe situations in v.»hich a Thurstone (equal appearing in- 
tervals) scale is more appropriate than a Likert (sununated 
ratings) scale, and vice versa, 

2,10 11, Construct a Tiiurstoiie scale for assessing attitude X, including 
computing the scale’s reliability. 

2,17 12, Construct a Likert scale for assessing attitude Y, including 
computing of the scale’s reliabilitj'^. 



2,41 13, Define a given academic problem in tenns of observable operations. 



2,35 14, Devise and conduct a Q sort to determine change in teachers ’ 
attitudes associated with a given program. 

2,64 15, I have a study involving upgrading curricular offerings, I 
can supplement my statistical data with e\,^idence from non 
reactive, unobtrusive variables. 
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Institute on Research' Design and Evaluation 
Indiana University 
Noven-ber 20-26, 1968 



Roster of Participants 



Name 


Address 


Cook, Ricliard Paul 


620 East 10th Place 
Gary, Indiana 46402 


DaFoe, Darrell 


Research Office 
Cedar Holmes Building 


Davis, Frank ?.!, 


120 E, V/alnut Street 
Indianapolis, Indiana 


Hannah, James M. 


250 Main Street 
^Vhiteland, Indiana 46184 


Jones, Gary E, 


4520 Y/. Ridge :>*139 
Gary, Indiana 46408 


Jordan, John G. 


Box 121 

Batesville, Indiana 47006 


Lane, Elizabeth ?.!. 


4390 Hyland Drive 
Dayton, Ohio 45424 


Linderman, Jim 


33621 Tav/as Tr, 

Westland, Michigan 43185 



Keekln, Korman Dale 959 Guilford Street 



Miller, Jerry L. 


Huntington, Indiana 46750 

1017 N, Sarah Street 
Mishawaka, Indiana 



Morgan , Bob 


200 N.W. 7th Street 
Evansville, Indiana 47708 

* 


Nev/ell, Ted F. 


1301 N. Michigan Street 
Plymouth, Indiana 46563 



Roster (cent.) 



Name 


Address 


Pruitt, Gene E, 


6901 Zionsville Road 
Indianapolis, Ind, 462S8 


Reichert, vraltcr 


1025 W, Rayen Avenue 
Youngs tov/n, Ohio 44502 


Richason, Eileen 


Box 303, R. R. i?2 
Logansport, Indiana 46947 


Roberts, Barbara 


112 ifoodla’.yn Avenue 
LaGrange, Kentucky 40031 



Robison, Del G, 


115 Harding Avenue 
llaryland Hts,, ?.Io, 63042 


Rowe, Robert B, 


R. R. //I 

Hope, Indiana 47246 


Sennett, A, Lowell 


8601 Colvjubia Avenue 
Munster, Indiana 46321 


Szyperski, Thoiaas 


1220 Howard Street 
Kalamazoo, Michigan 49001 


Tilley, Herbert T. 


R, R. #4, Box 224 
Goshen, Indiana 46526 


Ulsaker, Samuel 


290 V/, Michigan Ave. 
Jackson, Michigan 48201 

0 


V/atson, Janes H, 


Box 144 

Schererville, Indiana 

* * * * 


Dr. H. Glenn Ludlow 


School of Education, 253 
Indiana University 
Bloomington, Indiana 47401 


Dr, Clinton I, Chase 


School of Education, 253 
Indiana University 
Bloomington, Indiana 47401 


Dr, Richard C, Pugh 


School of Education, 253 
Indiana University 
Bloomington, Indiana 47401 



